PYNQ 506 - Genesis Robot Simulation¶

Learn to control a robot arm in 3D simulation!

By the end of this session, you will:

  • Control a robot arm using 3D coordinates
  • Perform pick-and-place operations
  • Understand inverse kinematics
  • Watch your robot in real-time via video stream

Real-World Applications¶

Companies like Tesla (assembly lines), Amazon (520,000+ warehouse robots), and NASA (Mars rover) use these skills every day!

What is Genesis?¶

Genesis Physics Simulation Server¶

Genesis is a powerful physics engine that simulates realistic robot movements:

  • Calculates gravity, friction, collisions
  • Figures out how to move 7 robot joints (complex math!)
  • Renders 3D graphics
  • Handles multiple students' robots at once

Why Client-Server?¶

Your Kria board is small and lightweight - perfect for sending commands. But running realistic physics is hard work and would make your board slow and hot!

Solution: Client-Server architecture

Your Kria (Client)          Genesis Server
"Move to [0.5, 0.0, 0.3]"  ───►  [Does complex math]
                            ◄───  "Done! Here's the result"

Why Use Simulation?¶

You might wonder: "Why not use a real robot?"

Simulation is essential in robotics development! Here's why:

💰 Cost

  • Real Franka Panda robot: ~$25,000
  • Simulation: Free!
  • Can test 20+ students at once without buying 20 robots

⚡ Speed

  • Test ideas instantly - no waiting for hardware setup
  • Run the same test 1000 times in minutes
  • Try crazy experiments without breaking anything!

🔬 Real-World Usage

Companies use simulation before deploying real robots:

  • NASA: Simulates Mars rover movements before sending commands (8-minute delay to Mars!)
  • Tesla: Tests factory robots in simulation before installation
  • Boston Dynamics: Trains robot dogs in simulation first
  • Self-driving cars: Waymo/Tesla simulate millions of miles of driving

The Process:

1. Design in simulation (safe, fast, cheap)
2. Test in simulation (find bugs early!)
3. Deploy to real robot (confident it works)

Simulation lets you fail fast and learn faster - without $25,000 repair bills! 🚀

In [ ]:
# Import the Genesis simulation client library
from pynqsim import SimulationClient
import time

# Server configuration
# Your instructor will provide the correct IP address
SERVER_IP = "172.20.166.171"  # Replace with actual server IP
SERVER_PORT = 9002           # Default Genesis port

# Connect to the server
sim = SimulationClient(SERVER_IP, SERVER_PORT)
print(f"Connected to Genesis server at {SERVER_IP}:{SERVER_PORT}")

# Video stream is available at http://{SERVER_IP}:9005
print(f"Video stream: http://{SERVER_IP}:9005")
In [ ]:
from IPython.display import HTML, display

def show_robot_video(title="Robot View"):
    """
    Display video stream automatically.
    Video is always visible - no clicking needed!
    """
    # Get the session token from the sim object
    token = getattr(sim, 'token', 'default')
    
    html = f'''
    <div style="
        border: 3px solid #2196F3; 
        border-radius: 8px;
        padding: 15px; 
        margin: 15px 0;
        background: #f5f5f5;
    ">
        <h3 style="margin: 0 0 15px 0; color: #2196F3;">📹 {title}</h3>
        
        <img src="http://{SERVER_IP}:9005/stream/{token}" 
             width="640" 
             style="max-width: 100%; border-radius: 4px; box-shadow: 0 2px 4px rgba(0,0,0,0.2);">
        
        <p style="margin: 10px 0 0 0;">
            <small>
                💡 <a href="http://{SERVER_IP}:9005/view/{token}" target="_blank">Open in new tab</a> for full-screen |
                Session: {token}
            </small>
        </p>
    </div>
    '''
    display(HTML(html))

print("Video helper function loaded!")
print("Use show_robot_video('title') to display the robot stream")

Create Robot Environment¶

In [ ]:
# Create a robot environment
# The "simple_cubes" scene includes:
#   - 1 Franka Emika Panda robot arm
#   - 1 table
#   - 3 colored cubes (red, green, blue)
sim.create_environment(scene="simple_cubes")

print("Robot environment created!")
print("Your robot is ready to control.")
print("Watch the video stream to see your robot in action!")

# Show live video feed
show_robot_video("Your Robot Environment")

Understanding 3D Coordinates¶

To control the robot, we need to tell it exactly where to move. We use a 3D coordinate system:

     Z (up/down)
     ↑
     │
     │       Y (left/right)
     │      ↗
     │     /
     │    /
     │   /
     │  /
     │ /
     |/_______________→ X (forward/backward)
    🤖 Robot Base

Three numbers tell the robot where to go:

  • X: How far forward (+) or backward (-) from the robot

    • Like steps forward/backward
    • Range: 0.3 to 0.8 meters

  • Y: How far left (-) or right (+) from the robot

    • Like steps left/right
    • Range: -0.3 to 0.3 meters

  • Z: How far up (+) or down (-)

    • Like height off the ground
    • Range: 0.0 (ground) to 0.5 meters

Units: Meters

  • 1 meter = 100 centimeters = about 3 feet
  • 0.5 meters = 50 centimeters = about 1.5 feet
  • 0.1 meters = 10 centimeters = about 4 inches

Example positions:

  • [0.5, 0.0, 0.3] = 50cm forward, centered, 30cm high
  • [0.4, -0.2, 0.1] = 40cm forward, 20cm left, 10cm high
  • [0.6, 0.15, 0.0] = 60cm forward, 15cm right, on the ground

Your First Robot Movement¶

Let's move the robot to a starting position! Watch the video stream as the robot moves.

In [ ]:
# Move robot to starting position
show_robot_video("Robot Moving to Starting Position")

print("Moving robot to starting position...")
sim.move_robot(0, position=[0.5, 0.0, 0.3], smooth=True, num_waypoints=50)
print("Movement complete!")
print("Robot is now at position [0.5, 0.0, 0.3]")
print("  X = 0.5 meters (50cm forward)")
print("  Y = 0.0 meters (centered)")
print("  Z = 0.3 meters (30cm above ground)")

Understanding move_robot()¶

sim.move_robot(0, position=[0.5, 0.0, 0.3], smooth=True, num_waypoints=50)
  • 0 = Your robot ID
  • position=[x, y, z] = Where to move (in meters)
  • smooth=True = Safe, smooth path (uses motion planning)
  • num_waypoints=50 = How smooth (higher = smoother but slower)

Checking the Robot's Position¶

We can ask the robot "where are you?" at any time using get_state().

In [ ]:
# Get current robot state
state = sim.get_state(0)

# Extract the end effector (hand) position
position = state['end_effector']['position']

print("Current robot hand position:")
print(f"  X = {position[0]:.3f} meters (forward/backward)")
print(f"  Y = {position[1]:.3f} meters (left/right)")
print(f"  Z = {position[2]:.3f} meters (up/down)")

What is sim.step()?¶

After each command, we call sim.step(50) to give the robot time to move.

Think of it like: Pressing the fast-forward button on a video!

  • Each step = 1/60th of a second
  • step(60) = 1 second of robot time
  • step(30) = 0.5 seconds

Gripper Control¶

What is the Gripper?¶

The gripper is the robot's "hand" - the end effector that can grasp objects. The Franka Panda has a two-finger parallel gripper:

  • Open position: Fingers spread apart (ready to grasp)
  • Closed position: Fingers together (holding an object)

The gripper is essential for pick-and-place tasks, which are fundamental to warehouse automation, manufacturing assembly, and many other robotics applications.

Opening and Closing the Gripper¶

Let's practice controlling the gripper. Watch the video stream to see the fingers move!

In [ ]:
show_robot_video("Gripper Control")

# Open the gripper
print("Opening gripper...")
sim.open_gripper(0)
sim.step(50)  # Give time for gripper to open
print("Gripper is now open")

# Close the gripper
print("Closing gripper...")
sim.close_gripper(0)
sim.step(80)  # Closing takes slightly longer
print("Gripper is now closed")

Pick-and-Place¶

What is Pick-and-Place?¶

Pick-and-place is the fundamental operation in robotics where a robot:

  1. Picks up an object from one location
  2. Moves it through space
  3. Places it at a different location

This seemingly simple task is the foundation of:

  • Warehouse automation: Amazon robots moving packages
  • Manufacturing: Assembly line robots installing components
  • Surgery: Robotic surgical systems manipulating instruments
  • Agriculture: Automated harvesting and sorting

The Standard Pick-and-Place Sequence¶

Every pick-and-place operation follows this pattern:

  1. Move ABOVE the object (safe height)
  2. OPEN the gripper
  3. Move DOWN to grasp height
  4. CLOSE the gripper (grasp object)
  5. Move UP (lift the object)
  6. Move to destination (above target location)
  7. Move DOWN to place height
  8. OPEN the gripper (release object)
  9. Move UP (move away safely)

Let's implement this step by step!

In [ ]:
print("=== PICK-AND-PLACE DEMONSTRATION ===")
print()
show_robot_video("Pick-and-Place Demonstration")

# Cube is at [0.5, -0.1, 0.02] on the ground
cube_x, cube_y, cube_z = 0.5, -0.1, 0.02

# Heights
safe_z = 0.35
hover_z = cube_z + 0.23
grab_z = cube_z + 0.11

# Destination
dest_x, dest_y = 0.6, 0.1

# ===== PICKING UP =====

# Approach
print("Step 1: Approaching cube...")
sim.move_robot(0, position=[cube_x, cube_y, hover_z], smooth=True, num_waypoints=100)
sim.step(30)

# Open gripper
print("Step 2: Opening gripper...")
sim.open_gripper(0)
sim.step(30)

# Descend
print("Step 3: Descending to grasp...")
sim.move_robot(0, position=[cube_x, cube_y, grab_z], smooth=False)
sim.step(75)

# Grasp
print("Step 4: Grasping cube...")
sim.move_robot(0, position=[cube_x, cube_y, grab_z], smooth=False)
sim.close_gripper(0)
sim.step(90)

# Lift
print("Step 5: Lifting cube...")
sim.move_robot(0, position=[cube_x, cube_y, safe_z], smooth=False)
sim.step(50)

# Secure grip
print("Step 6: Securing grip...")
sim.move_robot(0, position=[cube_x, cube_y, safe_z], smooth=False)
sim.close_gripper(0)
sim.step(120)

# ===== MOVING =====

# Midpoint
mid_x = (cube_x + dest_x) / 2
mid_y = (cube_y + dest_y) / 2

print("Step 7: Moving to midpoint...")
sim.move_robot(0, position=[mid_x, mid_y, safe_z], smooth=False)
sim.step(40)

print("Step 8: Moving to destination...")
sim.move_robot(0, position=[dest_x, dest_y, safe_z], smooth=False)
sim.step(40)

# ===== PLACING =====

# Lower
print("Step 9: Lowering to place...")
sim.move_robot(0, position=[dest_x, dest_y, grab_z], smooth=False)
sim.step(40)

# Release
print("Step 10: Releasing cube...")
sim.open_gripper(0)
sim.step(50)

# Move away
print("Step 11: Moving away...")
sim.move_robot(0, position=[dest_x, dest_y, safe_z], smooth=False)
sim.step(30)

print()
print("=== PICK-AND-PLACE COMPLETE! ===")
print(f"Moved cube from [{cube_x}, {cube_y}] to [{dest_x}, {dest_y}]")

Summary¶

Great work! You learned:

✓ Robot control with 3D coordinates [X, Y, Z]
✓ Pick-and-place operations - the foundation of warehouse automation
✓ Gripper control to grasp and release objects
✓ Inverse kinematics - the complex math behind robot movement
✓ Real-time visualization via live video streaming
✓ Video recording to capture and share your robot creations

These skills power Tesla factories, Amazon warehouses, and NASA rovers!

Challenges¶

Test your skills!

Tips:

  • Reset first: sim.reset() and sim.step(100)
  • Cube positions: Red [0.5, -0.1], Green [0.5, 0.0], Blue [0.5, 0.1]

Challenge 1: Move the Green Cube¶

Goal: Move the green cube to position [0.6, 0.15]

Modify the demo code above to pick up the green cube and place it at the new location.

In [ ]:
# Challenge 1: Move green cube to [0.6, 0.15]
sim.reset()
sim.step(100)

# Your code here:
# Modify the pick-and-place code to move green cube from [0.5, 0.0] to [0.6, 0.15]

Challenge 2: Stack Two Cubes¶

Goal: Stack the green cube on top of the red cube!

Hint: The second cube needs to be placed higher. Try grab_z + 0.04 for the place height.

In [ ]:
# Challenge 2: Stack green cube on red cube
sim.reset()
sim.step(100)

# Step 1: Move red cube to stacking location [0.5, 0.0]
# (if it's not already there)

# Step 2: Pick up green cube and place it ON TOP of red cube
# Hint: Place at higher Z position

Cleanup¶

When you're finished working with the robot, it's important to properly clean up your session. This frees up server resources for other students.

In [ ]:
# Destroy your robot environment to free up server resources
sim.destroy()

print("=" * 50)
print("Environment destroyed.")
print("Session complete!")
print("=" * 50)
print()
print("Thank you for exploring robotics simulation!")
print("Keep building, keep learning, keep creating!")